1. Field of the Invention
The present invention generally relates to multi-processor computer systems and more particularly to replacing a failing processor in a manner that is transparent to an operating system.
2. Description of the Related Art
In a computing environment, parallel processing generally refers to performing multiple computing tasks in parallel. Traditionally, parallel processing required multiple computer systems, with the resources of each computer system dedicated to a specific task, or allocated to perform a portion of a common task. However, recent advances in computer hardware and software technologies have resulted in single computer systems capable of highly complex parallel processing, through the use of multiple processors.
In some cases, a multi-processor system is logically partitioned, with one or more of the processors dedicated to, or shared among, each of several logical partitions. In a logically partitioned computer system, available system resources (including the processors, memory, and various I/O devices) are allocated among multiple logical partitions, each designed to appear to operate independently of the other. Management of the allocation of resources among logical partitions is typically accomplished via a layer of system firmware, commonly referred to as a partition manager.
An objective of the partition manager is to allow each logical partition to independently run software (e.g., operating systems and operating system-specific applications), typically developed to run on a dedicated computer system, with little or no modification. For example, one logical partition may be running a first operating system, such as IBM's OS/400, a second logical partition may be running a second operating system, such as IBM's AIX, while a third logical partition may be running a third operating system, such as Linux. By providing the ability to run multiple operating systems on the same computer system, a logically partitioned system may provide a user with a greater degree of freedom in choosing application programs best suited to the user's needs with little or no regard to the operating system for which an application program was written.
Logical partitioning of a large computer system has several potential advantages. For example, a logically partitioned computer system is flexible in that reconfiguration and re-allocation of resources may be easily accomplished without changing hardware. Logical partitioning also isolates tasks or groups of tasks, which may help prevent any one task or group of tasks from monopolizing system resources. Logical partitioning may also facilitate the regulation of resources provided to particular users, which may be integral to a business model in which the computer system is owned by a service provider who provides computer services to different users on a fee-per-resource-used or “capacity-on-demand” basis. Further, as described above, logical partitioning makes it possible for a single computer system to concurrently support multiple operating systems, since each logical partition can be executing in a different operating system.
Additional background information regarding logical partitioning can be found in the following commonly owned patents and patent applications, which are herein incorporated by reference: Ser. No. 09/672,043, filed Sep. 29, 2000, entitled “Technique for Configuring Processors in System With Logical Partitions”; Ser. No. 09/346,206, filed Jul. 1, 1999, entitled “Apparatus for Supporting a Logically Partitioned Computer System”; U.S. Pat. No. 6,467,007, entitled “Processor Reset Generated Via Memory Access Interrupt”; U.S. Pat. No. 5,659,786, entitled “System And Method For Dynamically Performing Resource Reconfiguration In A Logically Partitioned Data Processing System”; and U.S. Pat. No. 4,843,541, entitled “Logical Resource Partitioning Of A Data Processing.”
At times, a processor running in a multi-processor system, such as a logically partitioned system, may experience errors that are considered recoverable. An error is typically classified as recoverable if the error can be corrected with no loss of data. Once a processor has experienced a certain number of recoverable errors, it is probable that the processor is going to experience a non-recoverable (i.e., fatal) error in the near future. Hence, a recoverable error is also commonly referred to as a predictive failure (or predictive of failure). A non-recoverable error of a processor in a multi-processor system is a catastrophic event that leads to a check-stop condition in which all processors in the system are stopped, and an initial program load (IPL) of the system is performed with the failed processor persistently deconfigured (effectively removed) from the system.
An IPL (also referred to as a system boot process) generally refers to the process of taking a system from a powered-off or non-running state to the point of loading operating system specific code. This process could include running various tests on components and, in a multi-processor system all functioning processors would go through the IPL process, which may require a significant amount of time. Considering the fact that a logically partitioned system may include several partitions, a system IPL due to a check-stop condition caused by a processor failure may represent unacceptable downtime for the several partitions.
Therefore, in an effort to avoid check-stop conditions caused by fatal errors, it may be desirable to give up use of a failing processor when a number of recoverable errors are detected. Recoverable errors may be detected by a processor that is separate from the processors used to run operating systems, commonly referred to as a service processor. In a conventional multiprocessor system, the operating system can give up the use of the failing processor to prevent the occurrence of a check-stop condition. After giving up the failing processor, the operating system may then keep running or, if available, the failing processor may be replaced with a known good spare processor. The system may have one or more spare processors for a number of reasons. For example, in a capacity-on-demand system in which processing capacity is paid for by customers, there may be processors in the system which are not paid for by the customer (unlicensed processors). As another example, in a logically partitioned system, a (licensed) processor may not have been assigned to any partition.
FIG. 1 illustrates a flow diagram of exemplary operations for replacing a failing processor in a conventional multi-processor system. The operations of FIG. 1 are typically performed by a number of different components of a multi-processor system, including an operating system 112, system firmware 114, and a service processor 160. As illustrated, conventional replacement of a failing processor typically requires a number of complex interactions between the operating system and the system firmware.
The operations begin at step 12, when the service processor 160 detects a failing processor (e.g., a processor that has exceeded an allowable threshold number of recoverable errors). At step 14, the service processor 160 marks the processor to ensure the failing processor is not enabled on a subsequent IPL (such marking is commonly referred to as persistent deconfiguration). At step 16, the service processor 160 informs the system firmware 114 of the failing processor.
At step 18, system firmware 114 creates a log (for use by the operating system 112) to indicate a processor is failing and there is an available replacement. At step 20, the operating system 112 accesses the log (e.g., through a particular firmware call commonly referred to as an event scan) and detects the failing processor and the availability of a replacement. At step 22, the operating system 112 issues a firmware call to accept the addition of the replacement process. In response, system firmware 114 marks the replacement processor as assigned to the operating system 112, at step 24. At step 26, the operating system 112 issues a set of firmware calls to start the replacement processor under operating system control. At this point, the operating system 112 is now running with excess capacity, as the failing processor has not yet been given up.
Therefore, the operating system 112 takes a number of additional steps to deconfigure the failing processor from the operating system's point of view. For example, the operating system may remove timers associated with the failing processor, at step 28, inform other processors the failing processor is about to be stopped, at step 30, remove the failing processor from a global interrupt queue (a set of processors designated for interrupt handling), at step 32, and ensure no I/O device owned by the operating system 112 will send interrupts to the failing processor, at step 34. The operating system 112 then makes yet another series of specialized calls to system firmware 114 to stop the failing processor. In response, at step 38, system firmware 114 migrates all current workload away from the failing processor and prevents any new workload from being added, a process commonly referred to as dynamic deconfiguration (recall the failing processor was marked for persistent deconfiguration, at step 14, by the service processor 160, to ensure the failing processor is not restarted upon an IPL). Finally, having replaced the failing processor, the operations are terminated, at step 40.
This conventional technique for replacing a failing processor has a number of disadvantages. First, as illustrated in FIG. 1 and described above, the conventional technique requires multiple complex steps coordinated between the operating system and system firmware. Second, because of the required involvement of the operating system, each operating system must be separately designed to support processor replacement, which, for a logically partitioned system, runs counter to the objective of running different operating systems with minimal modification.
Accordingly, there is a need for an improved method and system for replacing a failing processor, preferably that requires little or no interaction on the part of the operating system.